[FEATURE] Add support of heterogeneous entities in batch renderer. by Kashu7100 · Pull Request #2960 · Genesis-Embodied-AI/genesis-world

Kashu7100 · 2026-06-17T00:21:33Z

Summary

Heterogeneous entities — objects whose geometry variant only exists in a subset of environments — were drawn in every environment by the Madrona batch renderer. This is the Madrona-path counterpart to #2958, which fixed the same class of bug in the pyrender rasterizer.

Root cause (and why it differs from #2958)

The pyrender fix in #2958 had two parts: (a) stop compacting per-env poses, and (b) add a per-(primitive, env) active_envs visibility mask.

The Madrona retriever (GenesisGeomRetriever.retrieve_rigid_state_torch) already feeds dense per-env poses — vgeoms_state.pos transposed to [num_worlds, num_vgeoms] — and never compacts them, so part (a) is already correct here. The only missing piece is part (b): per-environment visibility. Without it, a variant vgeom that exists only in envs X, Y is still drawn in all worlds (the Madrona renderer draws every geom in every world).

Change

GenesisGeomRetriever.retrieve_rigid_meshes_static now emits an optional geom_env_mask of shape [num_worlds, num_vgeoms] (1 = visible, 0 = hidden), built from each vgeom's existing active_envs_mask (None ⇒ active in all envs). This is the direct Madrona analog of the active_envs mask added to the pyrender rasterizer in #2958.

Omitted entirely for homogeneous scenes → preserves legacy behavior and stays back-compatible with Madrona builds that don't yet accept the kwarg.
No pose changes needed (the dense per-env poses are already aligned); inactive (world, geom) entries are simply masked out and never drawn / never enter the BVH.

if any(vgeom.active_envs_mask is not None for vgeom in vgeoms):
    num_worlds = max(self.rigid_solver.n_envs, 1)
    geom_env_mask = np.ones((num_worlds, self.n_vgeoms), dtype=np.int32)
    for vgeom in vgeoms:
        if vgeom.active_envs_mask is not None:
            geom_env_mask[:, vgeom.idx] = vgeom.active_envs_mask.detach().cpu().numpy()
    args["geom_env_mask"] = geom_env_mask

Dependency

Requires the matching renderer-side change in gs-madrona that consumes geom_env_mask: Genesis-Embodied-AI/gs-madrona#53 (adds the optional geom_env_mask kwarg → per-(world, geom) renderable disable). With an older gs-madrona, the mask is only emitted for heterogeneous scenes, so homogeneous workloads are unaffected; heterogeneous scenes need both PRs.

Testing / follow-up

py_compile clean; pre-commit (ruff) passes.
Functional validation needs a CUDA + Madrona environment. Suggested follow-up: a test_render-style test mirroring test_rasterizer_renders_heterogeneous_entities from [BUG FIX] Render each heterogeneous variant in its own environment. #2958 but driving the batch renderer (assert a variant only covers pixels in its active envs). Happy to add it.

Note

Draft — opened for review of the approach alongside gs-madrona#53.

🤖 Generated with Claude Code

Kashu7100 · 2026-06-17T01:31:48Z

Kashu7100 · 2026-06-17T01:32:10Z

video_old.mp4

video_new.mp4

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 12d4450d45

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Heterogeneous entities (objects whose geometry variant only exists in a subset of environments) were drawn in every environment by the Madrona batch renderer. Unlike the pyrender path fixed in Genesis-Embodied-AI#2958, the Madrona retriever already feeds dense per-env poses (`vgeoms_state.pos` -> [num_worlds, num_vgeoms]) and never compacts them, so the poses were already aligned -- the missing piece was per-environment visibility. `GenesisGeomRetriever.retrieve_rigid_meshes_static` now emits an optional `geom_env_mask` of shape [num_worlds, num_vgeoms] (1 = visible, 0 = hidden) built from each vgeom's `active_envs_mask`. This is the Madrona analog of the `active_envs` mask added to the pyrender rasterizer in Genesis-Embodied-AI#2958. The mask is omitted for homogeneous scenes, preserving legacy behavior and keeping back-compat with Madrona builds that do not yet accept it. Requires the matching gs-madrona change that consumes `geom_env_mask` (Genesis-Embodied-AI/gs-madrona#53). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…eneous-env

github-actions · 2026-06-17T20:21:57Z

🔴 Benchmark Regression Detected ➡️ Report

Kashu7100 · 2026-06-28T02:09:25Z

Now Genesis-Embodied-AI/gs-madrona#53 is landed.

The retriever must emit geom_env_mask matching each vgeom's active_envs_mask (correct shape/dtype/column alignment, each env seeing exactly one heterogeneous variant) and omit the key entirely for homogeneous scenes, since the args dict is splatted verbatim into the MadronaBatchRenderer constructor. Verified to fail with KeyError without the batch_renderer.py fix and pass with it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The heterogeneous per-env visibility mask is forwarded verbatim to MadronaBatchRenderer, whose constructor only accepts geom_env_mask from gs-madrona 0.0.8 (Genesis-Embodied-AI/gs-madrona#53); older wheels raise TypeError when building any heterogeneous scene with the batch renderer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Removed comment about gs-madrona version requirement.

Kashu7100 · 2026-07-07T04:25:05Z

gs-madrona 0.0.8 has been released.

…eous-env

…eneous-env

…neous render test The white-box test_heterogeneous_geom_env_mask_for_batch_renderer only inspected the geom_env_mask array the retriever emits; it never rendered, so it could not catch a break in the actual Madrona render path. Replace it with test_madrona_renders_heterogeneous_entities, an end-to-end regression that renders a heterogeneous entity (sphere in some envs, duck mesh in others) plus a homogeneous box through both Madrona backends and pixel-matches each per-env frame against a captured PNG snapshot (ground truth hosted on the HF snapshots dataset). It also asserts the homogeneous box renders in every env and that the duck variant does not leak into the envs it is not active in. Note: requires the ground-truth PNGs to be uploaded to the HF snapshots dataset and HUGGINGFACE_SNAPSHOT_REVISION bumped accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

duburcqa

Need a snapshot-based unit test. You should also remove 'test_heterogeneous_geom_env_mask_for_batch_renderer', we do not provide any guarantee regarding these internals.

Merge test_rasterizer_renders_heterogeneous_entities and test_madrona_renders_heterogeneous_entities into a single test_renders_heterogeneous_entities parametrized over RASTERIZER, BATCHRENDER_RASTERIZER and BATCHRENDER_RAYTRACER (mirroring test_render_api_advanced), removing the duplication and the misleading "madrona vs rasterizer" split. The rasterizer composites all envs into one image (env_idx == -1) while the batch renderer returns one image per env; the test branches on that but asserts the same per-env visibility for every backend: the homogeneous box renders everywhere, the duck variant never leaks into envs it is inactive in (masked out entirely when unbatched), and each frame pixel-matches its captured snapshot. Note: snapshot names changed with the rename/reparametrization; the GT PNGs must be (re)uploaded to the HF snapshots dataset and HUGGINGFACE_SNAPSHOT_REVISION bumped accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-13T14:05:52Z

🔴 Benchmark Regression Detected ➡️ Report

Kashu7100 changed the title ~~[Vis] Fix heterogeneous entities in the Madrona batch renderer~~ [BUG FIX] Fix heterogeneous entities in the Madrona batch renderer Jun 17, 2026

Kashu7100 marked this pull request as ready for review June 17, 2026 01:33

Kashu7100 requested review from YilingQiao and duburcqa as code owners June 17, 2026 01:33

chatgpt-codex-connector Bot reviewed Jun 17, 2026

View reviewed changes

Comment thread genesis/vis/batch_renderer.py

Kashu7100 force-pushed the fix/madrona-heterogeneous-env branch from 12d4450 to e3f8641 Compare June 17, 2026 01:39

Merge remote-tracking branch 'upstream/main' into fix/madrona-heterog…

4d4d3a7

…eneous-env

Kashu7100 and others added 3 commits July 4, 2026 02:08

Update pyproject.toml to remove unnecessary comment

67977e6

Removed comment about gs-madrona version requirement.

Kashu7100 and others added 3 commits July 7, 2026 00:27

Merge remote-tracking branch 'origin/main' into fix/madrona-heterogen…

da1dc85

…eous-env

Merge remote-tracking branch 'upstream/main' into fix/madrona-heterog…

7ce996c

…eneous-env

Milotrince mentioned this pull request Jul 10, 2026

[FEATURE] Per-environment geometry: heterogeneous variants, runtime scale, geometry pool, and ragged topology Milotrince/Genesis#13

Open

duburcqa requested changes Jul 11, 2026

View reviewed changes

Kashu7100 and others added 2 commits July 13, 2026 05:46

Update snapshot commit.

1d27e39

duburcqa changed the title ~~[BUG FIX] Fix heterogeneous entities in the Madrona batch renderer~~ [BUG FIX] Fix heterogeneous entities in the Madrona batch renderer. Jul 13, 2026

Merge branch 'main' into fix/madrona-heterogeneous-env

0e8c32f

duburcqa changed the title ~~[BUG FIX] Fix heterogeneous entities in the Madrona batch renderer.~~ [FEATURE] Add support of heterogeneous entities in batch renderer. Jul 13, 2026

duburcqa merged commit 2585e54 into Genesis-Embodied-AI:main Jul 13, 2026
19 of 20 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[FEATURE] Add support of heterogeneous entities in batch renderer.#2960

[FEATURE] Add support of heterogeneous entities in batch renderer.#2960
duburcqa merged 11 commits into
Genesis-Embodied-AI:mainfrom
Kashu7100:fix/madrona-heterogeneous-env

Kashu7100 commented Jun 17, 2026

Uh oh!

Kashu7100 commented Jun 17, 2026

Uh oh!

Kashu7100 commented Jun 17, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

Kashu7100 commented Jun 28, 2026

Uh oh!

Kashu7100 commented Jul 7, 2026

Uh oh!

duburcqa left a comment •

edited

Loading

Uh oh!

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Kashu7100 commented Jun 17, 2026

Summary

Root cause (and why it differs from #2958)

Change

Dependency

Testing / follow-up

Uh oh!

Kashu7100 commented Jun 17, 2026

Uh oh!

Kashu7100 commented Jun 17, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

Kashu7100 commented Jun 28, 2026

Uh oh!

Kashu7100 commented Jul 7, 2026

Uh oh!

duburcqa left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

duburcqa left a comment •

edited

Loading